How the CloudStrike Glitch Shut Down the Airlines and Crippled Global Commerce for a Day

Jon Healey
Los Angeles Times (TNS)

Talk about irony: The software that paralyzed Windows computers around the world late Thursday night and early Friday morning was planted by a company that protects Windows computers against malware.

That company is CrowdStrike, a publicly traded cybersecurity firm based in Austin, Texas. It acknowledged the problem around 11 p.m. Thursday and started working on a solution, offering a workaround in the wee hours Friday and a fix a few hours later.

The vast sea of Blue Screens of Death triggered by CrowdStrike’s error is a testament to the market-leading status of the company’s software, which detects and defends against malicious code planted by hackers. Its approach is known as “endpoint security” because it installs its defenses on devices that connect to the internet, such as computers and smartphones.

According to the website 6sense.com, CrowdStrike has more than 3,500 customers, which represent about one of every four companies buying endpoint security. Although most of its customers are based in the United States, it has hundreds in India, Europe and Australia, 6sense reports.

Here’s a quick explanation for how things went wrong so quickly for so many Windows users around the world, including airlines, hospitals, banks and government agencies.

The Falcon Sensor update

One of the selling points of CrowdStrike service is that it can improve its defenses rapidly as new threats are discovered. As part of that service, it continuously and automatically updates the Falcon Sensor software on its customers’ machines.

Automatic updates are, under normal circumstances, a good cybersecurity practice because they prevent clients from having machines with outdated defenses on their networks. But the latest incident reveals the flip side of the coin.

According to CrowdStrike, the problem was triggered by a “single content update” for its customers with Windows PCs. The buggy code wasn’t detected until after it had downloaded and installed on many of CrowdStrike’s clients machines.

Once loaded, the bad update interfered with core functions of the PC, causing Microsoft’s infamous blue error screen to pop up and convey a message along the lines of, “Your PC ran into a problem and needs to restart.” And as long as the update remained in place, restarting the machine led to the same errant result.

The fix offered by CrowdStrike

CrowdStrike stopped sending out the faulty update early Friday morning, so machines that had not loaded it yet were spared the turmoil.

For machines caught in the cycle of blue-screen hell, the company initially offered step-by-step instructions for how to reboot Windows in a mode that would allow them to find and delete the buggy update. The drawback, as many commenters online noted, is that this machine-by-machine approach isn’t much help for organizations with hundreds or thousands of bricked PCs.

According to the tech website 404, Microsoft also suggested rebooting a crashed machine multiple times— as many as 15 — could solve the problem.

Within a few hours, CrowdStrike was distributing a piece of software that removed the buggy code. This worked only for customers whose machines were able to connect to the internet and download the fix, though; everyone else would be left with the PC-by-PC workaround.

The lessons from the CrowdStrike debacle

Some Macintosh and Linux users, who were immune to the CrowdStrike-induced upheaval, devoted a portion of their morning Friday to spiking the football on Windows, even though the problem wasn’t caused by Microsoft.

Other observers argued that the incident demonstrated the risk of having one potential point of failure affecting millions of computers — a problem that has been demonstrated repeatedly during the broadband era.

Transportation Secretary Pete Buttigieg made a similar point at a press conference Friday in East Los Angeles. “A lot of people around the country and around the world are shocked to discover that a single issue with a single piece of software can have that many knock-on implications. So … that’ll be a question that really goes to the design of our systems for the long-term,” Buttigieg said.

“As a recovering computer science major,” Rep. Ted Lieu (D-California) said on X, “I’m not surprised a faulty update by CrowdStrike took down Microsoft Windows. Always risks in giving another software program full or near full access to an operating system.”

Steve Garrison, founder of Stellar Cyber in San Francisco, said it’s more important to figure out how to make improvements than to play the blame game. This incident, he said, underscores the need for companies to spend plenty of time checking the quality of their products in a controlled environment before releasing them to customers.

Another lesson, he said, is the need for companies, their competitors and their customers to work together as a community to spot problems. “What do we need to do to check the checkers of our supply chain?” he asked.

Dan O’Dowd, a developer of security software for the military, said the fiasco demonstrates that we need better software in critical systems.

“The immense body of software developed using Silicon Valley’s ‘move fast and break things’ culture means that the software our lives depend on is riddled with defects and vulnerabilities,” O’Dowd said in a statement. “Defects in this software can result in a mass failure event even more serious than the one we have seen today.”

He added, “We must convince the CEOs and Boards of Directors of the companies that build the systems our lives depend on to rewrite their software so that it never fails and can’t be hacked. … These companies will not take cybersecurity seriously until the public demands it. And we must demand it now, before a major disaster strikes.”

Small Business